3D versus 2D Pose Information for Recognition of NGT Signs

In this article, we evaluate the improvement in sign recognition that can be achieved with 3D tracking, compared to recognition with image plane tracking. Experiments are shown using a pair of stereo cameras, from which 3D positions of the segmented hands are estimated by triangulation between the left and right cameras. A sign classifier is trained on a set of 37 different NGT signs using 2D and 3D features. The results show that 3D features improve sign detection results, especially when 3D features are only used when their relevance is known beforehand. Furthermore, we show the positive effect of perspective distortion at close range.